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Focusing the Camera Lens on the Nature of Science: Evidence for the 
Effectiveness of Documentary Film as a Broader Impacts Strategy 
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ABSTRACT 

Scientists' involvement in education has increased in recent years due to mechanisms such as the National Science Foun¬ 
dation's "broader impacts" expectations for research projects. The best investment of their effort lies in sharing their exper¬ 
tise on the nature and processes of science; film is one medium by which this can be done efficiently. In this article, we 
illustrate this approach through the development of Upward and Outward: Scientific Inquiry on the Tibetan Plateau, a 20 min 
educational documentary film for school science classrooms and teacher professional development. The film portrays the 
intellectual and human processes of science as seen through the work of an international team of scientists on an interdis¬ 
ciplinary geoscience research project. Evidence gathered from pre/post classroom assessment responses by 350 students 
in grades 6-14 indicates that students absorb a variety of messages about the intellectual and social processes of science. 
These ideas contrast with their prior knowledge, counter common stereotypes of science and scientists, and broaden 
students' notions of the scientific method. The film aligns with national and state standards on scientific inquiry and the 
nature of science. ©2011 National Association of Geoscience Teachers. [DOI: 10.5408/1.3604825] 


INTRODUCTION: SCIENTISTS’ ROLE IN 
MODELING INQUIRY 

Leaders in U.S. science and education have called for 
scientists to participate in improving public understanding 
of science (Alberts, 1991, 1993, 2009; Bybee, 1998; Colwell 
and Kelly, 1999; Dolan et al., 2004; Fraknoi, 2005; Obama, 
2010). U.S. federal agencies that fund scientific research 
have provided incentives or stated expectations that scien¬ 
tists contribute to science education and outreach. For 
example, NASA's Science Mission Directorate requires that 
all missions have a "robust and substantial" commitment 
to education and public outreach activities (NASA, 2008, p. 
1). In reviewing research proposals, the National Science 
Foundation weighs the "broader impact" of research — 
which may include activities to educate and inform 
students, teachers, policymakers or the general public (e.g., 
NSF, 2003). We refer to these activities collectively as edu¬ 
cation and outreach (E/O), following Franks et al. (2006), 
where education is the teaching and learning of knowl¬ 
edge, skills, and cultural beliefs through formal (in school) 
or informal (self-directed, out of school) activities. Public 
outreach activities generate awareness and interest, and 
may also support education. 

These activities are motivated by concerns about pub¬ 
lic understanding of science. A majority of Americans are 
interested in science, but "do not give correct answers to 
questions about basic factual knowledge of science or the 
scientific inquiry process" (National Science Board, 2010, 
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p. 7-4). Widespread misconceptions about the nature of 
science and scientific knowledge contribute to contention 
rather than informed debate over issues such as climate 
change, stem cell research, and the teaching of evolution 
in schools (McComas, 1996). School textbooks may rein¬ 
force these myths by portraying "the scientific method" 
as a standardized, linear, four- or five-step process, in 
contrast with the complex, nuanced, and question-driven 
process described by scientists (Reiff et al, 2002; 
Harwood et al., 2002). Such portrayals address neither the 
inherent uncertainties in scientific knowledge, nor the 
iterative and social construction of that knowledge. These 
notions become problematic when they lead to over¬ 
reliance on science to provide definitive answers or, con¬ 
versely, to loss of confidence in science as never getting 
anything "right." 

Many people also hold stereotypes of scientists that 
may interfere with their understanding of science or their 
beliefs about its usefulness or reliability. One tool used to 
measure these ideas is the "Draw a Scientist" test (DAST) 
(Chambers, 1983; Finson, 2002; Sjoberg, 2000). Children's 
drawings made for the test often portray brainy, anti-social 
geeks who work alone in labs and explode things. These 
widespread views are corroborated by data from surveys, 
essays, and interviews (Finson, 2002); adults' ideas — 
including teachers'— are substantially the same (Rahm and 
Charbonneau, 1997; Barman, 1999; Cady and Rearden, 
2007). Evidently, people's notions of scientists do not 
evolve greatly over time despite education and other life 
experiences. These stereotypes are robust in society too: 
study results are largely unchanged over the decades 
(Mead and Metraux, 1957) and startlingly similar across 
cultures (Sjoberg, 2000). One small bright spot: children 
draw fewer representations of Frankenstein today than in 
1957. 

Goals for science education that address these percep¬ 
tions of science are well articulated in national guiding 
documents such as the National Science Education Stand¬ 
ards (National Research Council, 1995; see also American 
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Association for the Advancement of Science, 1990). The 
standards identify three broad learning goals for all 
students: 

1) To learn science —the big ideas and central concepts 
of science. For example, plants grow by using 
energy from the sun to convert carbon dioxide from 
air into biomass. 

2) To learn how to do science —the skills and habits of 
the mind required to conduct an investigation. 
These include both practical skills (measuring, 
graphing) and intellectual abilities (controlling a 
variable, interpreting a negative finding). 

3) To learn about science —the understanding that sci¬ 
ence is a human activity and a method of construct¬ 
ing knowledge. For example, scientists hold 
particular standards for which evidence and meth¬ 
ods are acceptable, which may differ from those of 
other disciplines. 

These two latter goals emphasize the scientific process 
for building new knowledge as distinct from the existing 
facts and concepts of scientific knowledge stated in the first 
goal (Olson and Loucks-Horsley, 2000). But these goals are 
related: as a recent report notes, "Developing deep under¬ 
standings of science requires understanding the nature of 
scientific explanations, models, and theories as well as the 
practices used to generate these products" (Bell et al, 2009, 
p. 275). 

In practice, most classroom time and textbook space 
address Goal #1, the big ideas of science. Goal #2, learning 
to do science, is most often addressed through laboratory 
work or hands-on inquiry. The generally poor state of labo¬ 
ratory instruction (Singer et al, 2006) and the growing use 
of inquiry curricula and instruction give reason for both 
despair and hope that this goal can be achieved. Goal #3, 
learning about science, is rarely addressed explicitly. Many 
instructors assume that students will glean these ideas as a 
side effect of laboratory work, albeit with little evidence to 
support that belief. 

Goals #2 and #3 are related but distinct: A students' 
ability to perform an investigation should be better if they 
recognize general qualities of a good or bad investigation. 
But it is challenging to abstract from a particular laboratory 
experience to general understanding. Even if students can 
interpret what they have learned when a white powder 
does not react with hydrochloric acid, this seldom yields a 
broader understanding about negative evidence that can 
be applied to other problems. These are hard topics for 
teachers too. Few science teachers have a chance to 
develop personal knowledge of the process of science by 
conducting research first-hand. In their own schooling, 
most experienced fact-packed lecture courses rather than 
deep engagement with gathering, interpreting, and com¬ 
municating about scientific evidence. Teachers are thus at a 
disadvantage in teaching about the nature of science, and 
few curricular resources directly address these ideas. 

Given these gaps in understanding and in school pro¬ 
gramming, it is especially crucial for scientists to share 
their expertise on the nature and process of science. These 
concepts are not discipline-specific: every scientist is an 
expert who can offer rich examples of science at work in 
her own field. We have elaborated this point elsewhere 
(Laursen, 2006; Laursen and Smith, 2009; Thiry et al., 2008). 


Here, we describe one way that scientists and their collabo¬ 
rators can communicate the nature and process of science 
to students and teachers, through an educational docu¬ 
mentary film for secondary students and teachers. Film 
can reach sizable audiences in and out of classrooms and 
leverages scientists' time efficiently. As curriculum, film is 
a flexible, useful, and familiar tool for teachers. Indeed, 
other geoscience research projects have used film or inter- 
net-delivered video as an outreach medium, including 
such examples as ANDRILL's Project Iceberg (ANDRILL, 
2007-2009), the Integrated Ocean Drilling Project (IODP- 
MI, 2005), and EarthScope (Prose and LaMacchia, 2006). 
Like our film, these touch on important aspects of geosci¬ 
ence such as field work and interdisciplinarity. But Upward 
and Outward differs significantly in making the nature and 
work of scientists its explicit and central theme, rather than 
an implicit aside within a content-focused presentation. 
We present evidence below that this approach succeeds in 
bringing these ideas to the fore of viewers' minds. We 
describe the design and use of the film, offer evidence 
about what students learn from the film, and suggest 
lessons learned here that can be applied to other science 
E/O efforts. 

THE DOCUMENTARY FILM AND ITS 
DEVELOPMENT 

To address the paucity of classroom materials avail¬ 
able for teaching about science, Laursen and colleagues 
developed a short educational documentary film and dis¬ 
tributed it to secondary school educators. Funded as a 
"broader impact" component for a large collaborative 
research study in geoscience, the film set out to: 

• Portray both the intellectual process of scientific in¬ 
quiry — posing and investigating questions — and the 
social process of science as a human endeavor. 

• Show science as a profession with places for people 
of varied backgrounds, skills and interests, doing a 
variety of jobs. 

• Offer examples of science involving observations as 
well as experiments; field work, computer modeling, 
invention, and laboratory work. 

• Depict science as collaborative and interdisciplinary. 

• Draw parallels between how the scientific team 
approaches a research project and how students con¬ 
duct investigations. 

The educational documentary film, Upward and Out¬ 
ward: Scientific Inquiry on the Tibetan Plateau, is centered on 
a multidisciplinary, collaborative, research project to study 
the geologic uplift of the Tibetan Plateau and its impact on 
atmospheric and climate processes. The research team 
includes scholars and students in geology, geophysics, 
geochemistry, paleoclimate, and meteorology. They use 
geophysical methods, mapping, and age dating of rocks to 
understand how and when the plateau grew, both rising in 
altitude and expanding to the north (e.g., Molnar and 
Stock, 2009; Dayem et al, 2009). Isotopic analysis of basin 
sediments and meteorological modeling are used to deci¬ 
pher past changes in wind and precipitation patterns that 
occurred as the plateau grew and began to interact with 
the jet stream (e.g.. Roe, 2009; Dayem et al, 2010). A critical 
intellectual step is to compare the timing of these events: 
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Can these regional climate changes be linked to the 
plateau's upward and outward tectonic growth? (Molnar, 
2005). 

Rather than explaining the scientific content of the 
research, however, the film focuses on the process of sci¬ 
ence as seen through this project. Thus, it is appropriate for 
any science class, not just Earth science. Viewers follow the 
international team of scientists as they work in the lab and 
in the field, build new instruments and computer models, 
travel to China for field work and conferences, argue about 
their findings, and enjoy collaboration and conversation. In 
gaining an insider's glimpse into both the intellectual 
process of scientific inquiry and the everyday social and 
professional activities of science, students learn how sci¬ 
ence is a human process for building knowledge, not just a 
body of fact, and they come to see scientists as real people. 

Originally targeted to students in grades 8-10, the film 
has proven effective with both older and younger students, 
from sixth-graders to undergraduates and teachers. At 20 
min, the film can be readily screened and discussed within 
a single class session. A web site (http://cires.colora- 
do.edu/education/ outreach/ TibetOutwardUpward/) 
provides teachers with a description of the film and its 
links to the inquiry standards, suggested pre/post writing 
prompts, discussion questions, teaching tips, and back¬ 
ground on the film's educational and scientific content, 
and students' likely prior conceptions. This is not a pre¬ 
scriptive lesson plan, but a set of practical ideas to help 
teachers place the film in their own curricular context, 
adapt its use to different age groups and learning objec¬ 
tives, and foster student engagement with its ideas. The 
film does not explicitly espouse inquiry or any other peda¬ 
gogical approach, but by showing how scientists think and 
work, we hope to encourage students and teachers to take 
a similar investigative approach to their classroom work, 
in line with Bower's (1996) admonition: 

Real science, whether in the laboratory or the 
classroom, depends substantially on the application of 
good scientific process. By scientific process I do not 
mean the famous four steps in the scientific method 
that are drilled into the heads of children from Grade 3. 
Instead, I mean the real scientific skills of investigation, 
critical thinking, imagination, intuition, playfulness, 
and thinking on your feet and with your hands that are 
essential to success in scientific research. 

Constructing the film required the scientific, educa¬ 
tional, and artistic expertise of many contributors. Funded 
through a National Science Foundation grant to the 
research team, the size and scope of the research project 
(5 years, 7 Pis, $2.9 million budget) warranted a significant 
material commitment to a specific "broader impacts" com¬ 
ponent. The lead investigator publicly stated his belief that 
including the film in the grant proposal as a specific E/O 
effort was a positive factor in receiving an award after a 
previously declined proposal. 

The concept was developed by author Laursen, an out¬ 
reach scientist with a Ph.D. in chemistry and experience in 
inquiry-based teaching and teacher professional develop¬ 
ment. Roslyn Dauber, an independent documentary film¬ 
maker, contributed her technical and artistic expertise; her 
background in science policy and interest in science were 


also assets. Together, Laursen and Dauber developed the 
story line and script. Scientific team members gave on- 
camera interviews and laboratory tours to explain the pro¬ 
ject and describe the scientific process; several provided 
advice, graphics, animations, and still images. One gradu¬ 
ate student carried a video camera to China and shot foot¬ 
age for the film; another shared stunning photographs 
from his time in the field. 

Laursen worked with individual scientists and 
attended team meetings to learn about the research. When 
a rough cut (draft) was ready, she screened it to gather 
scientists' input. It was important to us to get the science 
right, but we experienced a steady tension between accu¬ 
racy and simplicity. Some scientists felt the film offered too 
shallow a treatment of their subject, and indeed it was 
tempting to explain interesting details about the study or 
its methods. We found it essential to frequently restate our 
goal to communicate about how science works, and to 
clarify why that was important. As the film progressed, 
we critically analyzed each cut for which elements best 
accomplished this goal within our rigorous length criterion 
of 20 min. 

Local teachers were recruited to respond to an early 
cut of the film; it was substantially revised based on then- 
feedback. For example, based on concerns about visual 
appeal to students, we added high-energy production fea¬ 
tures: animations, rapid cuts, mood-setting music, and a 
whimsical opening using vintage footage of scientists. 
With help from the director of a summer science camp, we 
incorporated footage of high school students doing science 
to more explicitly link the processes that scientists use in 
research and students use in the classroom. These efforts 
paid off, as one high school teacher reported: "It was a per¬ 
fect length, and [the] music and changing scenes kept then- 
attention." Thus, teacher ideas were critical in helping to 
improve the film; the features they suggested are also those 
shown by research to foster greater attention and learning 
(see Mares et al, 1999; Fisch, 2004). 

CLASSROOM REACH AND IMPACT 
OF THE FILM 

Distribution 

Based on teacher input and our experience in K-12 
classrooms, we produced the film as hard-copy DVDs. 
Most teachers have access to, and are comfortable using, a 
classroom video player; fewer feel well prepared to use the 
Internet in teaching (Russell et al., 2003; National Center for 
Education Statistics, 2000). We considered online distribu¬ 
tion but were concerned that the large file size required 
would tax school bandwidth: one failed download would 
deter a teacher forever, compared with the robust and 
familiar technology of a DVD player. Moreover, schools 
with more minority and/or low-income students tend to 
have poor access to classroom technology and make less 
use of it (National Center for Education Statistics, 2000; 
Owens et al, 2007). Cost posed an additional practical 
issue. Background music was chosen from a library of low- 
cost music clips offered by a licensing service; the fees for 
music rights are based on the number of viewers reached. 
The fees for web dissemination are high because a web- 
based movie hypothetically reaches all six billion people 
on the planet. 
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We distributed the film primarily through in-person 
contact, giving DVDs to educators who attended a screen¬ 
ing with the reasoning that this would increase classroom 
use compared to unsolicited mailings or conference booth 
giveaways. We showed and discussed the film at teacher 
workshops, education conferences, schools, and public 
science events, and gave copies to school district science 
leaders and educators from higher education, informal sci¬ 
ence, and research organizations who lead in-service 
courses for K-12 teachers. Through these colleagues, the 
film penetrated entire school districts and reached work¬ 
shops and conferences around the U.S. and in South Africa 
and Australia. After showings to scientists at conferences 
and local laboratories, some took DVDs for their own out¬ 
reach work or for friends and relatives who were teachers. 
Other educators requested the film via our web site. DVDs 
packaged in a hard plastic case could be mailed by apply¬ 
ing an address label and fourth-class postage. At screen¬ 
ings and on the web site, we invited teachers to share their 
students' responses to the film. Author Brickley, a science 
educator and professional developer who used the film 
and distributed it to teachers, analyzed the student 
responses thus gathered; this analysis is discussed below. 

Because we distributed the DVDs in person, we could 
track their distribution. To date, some 600 copies of the 
film have been distributed. Of these, over 60% went 
directly to educators: 31% to classroom teachers, 23% to 
professional developers who work with teachers, and 8% 
to college science instructors, including those teaching pre¬ 
service teachers. The other copies were distributed by the 
research team members to similar audiences, but we did 
not document their individual recipients. 

Teacher Response and Classroom Use 

Based on our observations at screenings, anecdotal evi¬ 
dence, and some evaluation data, it is evident that seeing 
and discussing the film themselves helped teachers under¬ 
stand its messages and recognize potential classroom uses. 
Indeed, we found the film to be particularly effective to 
kick off a session on teaching science through inquiry in 
stand-alone workshops or multiday professional develop¬ 
ment courses. Seeing scientists in action gave teachers a 
model for how their students could think and work like 
scientists in the classroom: posing questions, gathering evi¬ 
dence, and interpreting it; collaborating and making and 
critiquing arguments about their conclusions. Framing 
classroom inquiry as analogous to scientific research 
seemed to appeal to teachers as science professionals and 
could overcome some resistance to inquiry as the latest 
ed-school fad. 

Teachers also gained an enhanced understanding of 
the scientific process themselves. In a discussion after a 
screening, teachers consistently commented on collabora¬ 
tion as an aspect of the film that was surprising to them. 
They remarked on the film's portrayal of how diverse skill 
sets and perspectives helped to solve hard problems and 
how articulating ideas aloud to others could refine and 
strengthen them. They noticed that the scientists seemed to 
enjoy working together. One middle school teacher wrote: 

"I really like the collaboration aspect shown in the 

video —lots of different kinds of scientists from lots of 

different areas and institutions. The scientists all look 
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like real people, some in the lab and some in the field. 
It makes science look fun and exciting —which it is, but 
I think with all the testing we do we are taking the fun 
out of it!" 

A district science curriculum director emphasized why 
it was important to teach students about how science 
works, referencing parental skepticism of science taught in 
schools: 

"As a leadership group, we have been talking about 
the importance of teaching the nature of science and 
having our students understand the work scientists 
do. More and more we are faced with students 
requesting to opt out of certain parts of the curriculum 
(most often evolution during biology, but this year 
also diseases and climate change). We need to educate 
our students to think critically about the information 
they come across, whether in class, online, [or] in the 
movie theatre." 

Teacher comments also elucidate how the film reached 
student audiences. At closed-enrollment workshops where 
we could gather evaluation data, teacher responses to 
open-ended survey questions indicated their intent to 
make "immediate classroom use" of the film to "show 
what a scientist is" and "explain the scientific process." 
"It's so connected to what I am doing in the classroom 
now," wrote one. K-12 teachers reported using the film in 
biology, chemistry, Earth science, and general science 
classes across the middle and high school grades. "The 
video really opened their eyes to the many opportunities in 
the field. My 7th and 8th graders have a very narrow view 
of what you can do with what I am teaching them and 
how many jobs in this world are related to science," said 
one teacher. Higher education users were most likely to 
use the film in science courses for nonscience majors or for 
preservice teachers. "I reordered my class to start with a 
section on what is a scientist and what does it mean to 'do 
science' and an activity on observations and inferences. 
The video worked well [as part of this]," wrote one college 
instructor. Lesson ideas shared with teachers emphasized 
discussion, to foster learning through social interaction 
over the media experience (Kozma, 1991). 

From teacher reports, we can estimate the number of 
students reached by the 600 copies distributed to date. Ten 
classroom teachers sent us a total of 350 student papers. 
Thus, for this group, each DVD reached 35 students on 
average in one academic year. This estimate is likely low, 
because teachers did not necessarily send student papers 
for all class sections that viewed the film. With the conserv¬ 
ative assumption that only 50% of all DVDs distributed 
were shown to students, a total of 10,500 students (300 
DVDs x 35 students) viewed the DVD in one academic 
year. The actual number of students reached will be much 
larger if teachers show the film in multiple sections, 
courses, or semesters, and use it again in subsequent years. 

Evidence of Student Learning 

Qualitative data were collected to assess student learn¬ 
ing from the film. At teacher workshops using the film, 
teachers used a pre-/post-assessment tool that they could 
also use with students. While the main purpose of the 


130 S. Laursen and A. Brickley 


J. Geosci. Educ. 59, 126-138 (2011) 


assessment was to focus students' attention and foster 
classroom discussion, teachers were also invited to return 
classroom sets of anonymous student papers, which we 
used to assess the impact of the film on students. To 
acknowledge their help, participating teachers received a 
$10 gift certificate to an online bookstore. We asked teacher 
to conduct the assessment as modeled in the workshop. 
Before watching the video, students were asked to "draw 
or write a paragraph describing a scientist doing science." 
(Question 1). Following the video, students were asked to 
respond to two more questions: 

2. Describe something that surprised you in the DVD 
or gave you a new insight about doing science. 

3. Study Figures 1 and 2 on the next page, which show 
two different views of the scientific process. What 
elements of each did you see in the film? Which 
version do you think matches the film best? 

Figures 1 and 2 depict the same figures used with stu¬ 
dents in question 3. Figure 1 shows a mostly linear flow 
chart depicting the classic scientific method, similar to 
illustrations often found in school textbooks (Reiff et al., 
2002; Blachowicz, 2009) or in a simple internet query for 
"scientific method." It emphasizes the formation, testing, 
and validation of hypotheses. Figure 2 shows the "inquiry 
wheel" of Reiff et al. (2002), a model that was developed 
from interviews with scientists and which depicts a less lin¬ 
ear, more iterative view of science (see also Harwood, 
2004). This model includes activities such as observing, 
investigating the known, reflecting on findings, and com¬ 
municating results, in addition to hypothesis formation 
and testing. The entire assessment, with images, is 
available on our web site (http://cires.colorado.edu/ 
education/ outreach/ TibetOutwardUpward/ images / Student 
Work Survey.pdf) 

Altogether, 350 student responses were received from 
10 different classroom groups, most from Maine and Colo¬ 
rado: 5 groups of middle school (MS) students totaling 151, 
4 groups of high school (HS) students totaling 171, and 1 
group of 28 undergraduates (UG). Middle school students 
included students in grades 6-8 taking Earth science or 
general science. High school students were a mix of Earth/ 
Space science students and biology students in honors. 
Advanced Placement (AP), and mainstream classes. The 
undergraduates were taking a natural science course for 
nonscience majors. In this analysis, we have grouped the 
undergraduate students with the high school students. To 
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FIGURE 2. The inquiry wheel: Another view of the 
scientific process (as labeled for question 3). 


reduce the reporting burden on teachers who voluntarily 
administered the assessment and mailed student 
responses, and to maintain strict student anonymity, no 
data were collected on class sizes, content coverage, time 
of year, or on student gender, ethnicity, or socioeconomic 
status. 


METHODS OF ANALYSIS 

Using thematic content analysis (Libarkin and 
Kurdziel, 2002; Smith, 2000), student responses to each of 
the three questions were inductively coded into common 
themes. Specific words, phrases, or items included in the 
drawings were the markers for certain codes. Others were 
defined by the central idea presented. Some students' 
responses were coded with multiple themes, therefore 
when reporting counts we refer to the number of "ideas" 
or "observations" rather than the number of students 
discussing a particular issue. 

Five dominant themes emerged from analysis of ques¬ 
tion 1 and are listed in Table I, below. This first question 
was similar in nature to the commonly used Draw-A- 
Scientist Test (DAST) of Chambers (1983), but broader 
themes and themes based on word cues or descriptions 
were used rather than the specific criteria used by Cham¬ 
bers or the Draw-A-Scientist Checklist, DAST-C, devel¬ 
oped by Finson, Beaver, and Cramond (1995). Question 2 
asked students to describe something that surprised them 
in the DVD or gave them new insight into doing science. 
The eight dominant themes derived from coding are listed 
in Table II, below. Question 3 was analyzed at a more basic 
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FIGURE 1. Traditional view of the scientific method (as labeled for question 3). 
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level because many students did not completely answer 
the question as phrased. Most students did identify which 
diagram they thought was best represented in the movie. 
Responses were coded as selecting Fig. 1 (Classic Scientific 
Method), Fig. 2 (Inquiry Wheel), or both; student reasoning 
was not coded because it was seldom given in detail. 

Once student responses were categorized by themes, 
we quantified the number of observations under each 
theme. One student paper may include multiple observa¬ 
tions about several themes. Most responses were brief, 
averaging 1.2-1.4 distinct observations per student for 
questions 1 and 2; high school and college students typi¬ 
cally reported more observations than did middle stu¬ 
dents. Because question 3 forced students to choose 
between two alternatives, we categorized only one obser¬ 
vation per student in that analysis. Below we quote student 
comments as examples of the themes found in coding. We 
have corrected some grammatical and spelling errors that 
were distracting, but left others to preserve the authenticity 
of students' voices. Quotations are labeled with the writer's 
age group and science class. 

FINDINGS 

Student Prior Conceptions 

In classroom use, question 1 served to surface student 
preconceptions before viewing the film. In our evaluation 
of the film's impact, we used it to ascertain that the prior 
conceptions of students about "scientists doing science" 
were in fact similar to those previously demonstrated in 
other DAST studies. Table I summarizes the five categories 
and proportions of student observations in each. 

While based on a coding scheme that was independ¬ 
ently developed, our results fell largely in line with the 
stereotypes expected from DAST studies. Of the 350 stu¬ 
dents surveyed, fully two thirds identified scientists doing 
science with some feature of a "lab" or "laboratory," 
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accounting for 43% of observations. Many of these were 
drawings of lab tables and equipment; a written 
example is: 

Scientists think of expirements (sic) to do and create 
hypothesis's (sic) for what they believe will happen. 
Then they do the experiment and record observations. 
When the lab is over, they go over their notes. The 
experiments they do are based on what they want to 
learn. (HS AP biology student) 

Another 19% of the observations noted the unnatural 
appearance of scientists (e.g., wild hair). Middle school stu¬ 
dents (78%) more often identified science through lab 
equipment while high school and college students (37%) 
more often described the scientist's appearance. Drawings 
illustrated images like that presented in this student 
description: 

Scientists wear a big white lab coat, and they have 
goggles around their head. He has kind of curly/poofy 
white hair and glasses. He wears a suit underneath his 
lab coat. He has on fancy black laced up shoes. He's 
really pale looking and very old. Big, bushy eyebrows 
and mustache maybe. (HS chemistry/biology student) 

Another 13% of observations included the word 
"experiment/s" in the description, presenting experiments 
as a way to learn about the unknown: a trial with unex¬ 
pected outcomes. Use of this word was often nonspecific, 
as in this student response: 

When scientists do science I think that sometimes they 
don't know what they are doing and they are trying 
new stuff at the same time. So they experiment with 
things. (6th grade science student) 


TABLE I: Categories and frequency of student prior conceptions of scientists, from 350 student responses to question 1. 


Category 

Description 

Response type 
D = drawing, 
W = written 

Number of 
observations 

Percentage 
of all 

observations 

Similar 
categories in 
DAST-C (Finson et al., 
1995) 

Chemistry/ 

laboratory 

Specific mention of "chemi-"; notion 
that science is done in a lab. Drawings 
of someone holding beakers, flasks, etc. 

D&W 

234 

43% 

Symbols of research, sci¬ 
entists working in lab, 
symbols of knowledge 

Experiments 

Includes the word "experiments" — all 
verbal. Many double-coded with chem¬ 
istry/lab or observations 

W 

70 

13% 


Classic scientific 
method 

Includes words such as "observations," 
"hypothesis," "collecting data," 
"conclusions." 

W 

92 

17% 


Appearance 

Mostly drawings: wild hair, crazy 
expressions, Einstein-like images. Also 
comments on appearance. Does not 
include lab coat and glasses which are 
routine in "lab safety" education. 

D 

104 

19% 

Facial hair, mythic ster¬ 
eotypes, open comments 
on appearance 

Science in society 

Connections of science to other subject 
areas; the broader scope of science and 
its role in society, e.g. medical cures, 
new chemicals, use of mathematics. 

D&W 

45 

8% 

Relevant captions 
(formulas, idea bubbles) 

Total 



545 

100% 
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Other observations were more detailed, such as: "A 
scientist doing science is when someone is experimenting 
different hypotheses and theories" (HS Earth/space science 
student). Seventeen percent of observations included some 
feature of the classically taught scientific method (e.g., 
observe, hypothesize, collect data). Some classes had prob¬ 
ably covered this recently, as student descriptions were 
presented stepwise and described a procedure akin to the 
classical textbook presentation. 

A scientist when doing science has many things to do. 
They first come up with why they are going to do that 
topic. They then gather data/info and come up with a 
hypothesis (Ex: my guess is the experiment will work 
without glitches.) They will do the tests/trials of the 
experiment more than once or twice. They will have a 
conclusion about their topic. (MS science student) 

A small proportion (8%) of student observations were 
less stereotyped, making broader connections to the role of 
science in society and its links to other areas of research 
(i.e., medicine, use of mathematics). Some observations in 
this category were general, like this one: 

Science is whatever you decide to apply working and 
validated logic and concepts to answer or explain a 
question. A scientist is simply someone who decides 
he or she should ask the right questions to get the 
temporarily correct answer. (HS Earth/space science 
student) 

Others mentioned different fields or applications of 
science, as this sixth-grader shared: 

There are lots of different kinds of scientists]. Some 
of them mix things like chemicals together. Like 


Pharmists (sic), I consider them scientists. They mix 
different pills and medicines together. Some look at 
the earth and study the land and look at dirt and soil 
and see how it forms. They also study things like 
solids, liquids, and gases. (6th grade science student) 

These results set the stage, identifying our audience as 
students who hold a largely classical and stereotypical 
view of science as laboratory work done in seclusion by 
out-of-the-ordinary individuals. They are somewhat influ¬ 
enced, at least in terminology, by an introduction to the 
classical scientific method and to experimentation as a 
major way of doing science. Specific science topics, per¬ 
haps recently studied, appear in some answers. 

Student Change of Conceptions 

In their responses to question 2, completed after view¬ 
ing the film, students showed a notable change in their 
ideas, giving answers that exhibited a broader view of sci¬ 
entists doing science. The question asked them to 
"Describe something that surprised you in the DVD or 
gave you a new insight about doing science." In classroom 
use, this question aimed to have students reflect individu¬ 
ally on what they learned and generate ideas for class dis¬ 
cussion. Of the eight themes identified in coding, seven 
indicated that students were learning something "about 
science," including the nature of work as a scientist, how 
complete data collection is carried out, the background 
research and understanding behind good scientific ques¬ 
tions, and the necessary collaboration between scientific 
fields. Their language and remarks can sometimes be tied 
to very specific scenes or dialogue in the film. These 
themes and their frequency in student papers are detailed 
in Table II. 


TABLE II: Categories and frequency of changes in student conceptions of science and scientists, from 350 student responses to 
question 2. 


Category 

Description 

Number of 
Observations 

Percentage of all 
Observations 

Learning about science 



Method 

How much work is involved; how samples are taken; 
details, organization, time 

103 

24% 

Collaboration 

Collaboration of many different types of scientists from 
different specialties or countries; working together as a 
team; argumentation and discussion; friendship 

94 

22% 

Where 

Scientists travel to do their work, work outside 

54 

13% 

Question 

Specific reference to science as a process to answer any 
question 

40 

10% 

Equipment 

Engineering, invention of unique equipment 

36 

9% 

Fun 

People having fun, enjoying their work, being imaginative 
and creative 

22 

5% 

People 

Who is doing science: observations on gender, race, 
nationality of scientists; scientists' dress or appearance 

12 

3% 

Subtotal, learning 
about science 


361 

96% 

Learning science 

Science content learned, esp. formation of Plateau by 
colliding plates, learning Earth's history from rocks 

60 

14% 

Total 


421 

100% 
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Nearly a third of all students (representing 24% of all 
observations) expressed surprise at how science was car¬ 
ried out in the film, through shots of the field work, labs, 
samples, and explanations of what was done to gather 
evidence. Many students noted the number of samples col¬ 
lected, like this student: "I did not know that scientists 
took so many samples of rock, dirt, sand, etc." (HS Earth/ 
space science student). Another shared how his or her 
ideas had changed about what scientists do in a lab, "They 
don't just mix stuff.... They collect things for evidence." 
(6th grade science student). Some students gained a better 
understanding of what it means to form a hypothesis: "It is 
surprising that there is a lot of reading and research 
involved in making an hypothesis" (sic) (HS Earth/space 
science student). 

Middle school students were most struck by seeing 
scientists working together as a team, with 31% of 151 
students in this age group noting collaboration. Nearly a 
quarter (24%) of the high school and college students were 
also struck by the collaboration they witnessed in the 
video. These numbers suggest that neither age group had 
previously thought about scientists working together, and 
that the film challenged a common idea of scientists work¬ 
ing alone. 

There were different scientists] sharing different 
information w[ith] each other. Also there are field 
[and] indoor scientists]. (MS Earth science student) 

I'm surprised that scientists like working together. It 
seems like they would disagree and argue about 
everything. But I guess that just matters on who is in 
your group. (HS biology student) 

Another group of responses described science as "a 
way to answer any question." Ten percent of responses 
noted this idea, showing a widening view of how science 
is a thinking process and not just subject matter. This point 
was made explicitly in the film multiple times; stated by 
one scientist near the end of the film, the idea seemed to 
grab a subset of students. Some understood the process 
explicitly, as in this quotation: 

If you want to know something and you go out to find 
the answer to that question, you are a scientist. (HS 
Earth/space science student) 

Others took the idea generally, and may still hold 
some misconceptions, like this middle school student: 
"That if you can find something that you don't know what 
it is and you find out, that's science." (MS science student) 
The film showed scientists traveling the world and 
working in the field. This aspect of science surprised stu¬ 
dents and was noted in 17% of the observations. Their 
image of scientific work was expanded beyond lab work, 
as this middle school student expressed: "Something that 
surprised me in the DVD is when the scientists travel and 
not only stay in the room and work, they go to other 
states." (sic) Similarly, a high school student mentioned 
"That science isn't all about being in a lab, but it's every¬ 
where." (HS chemistry/biology student) 

We were also struck by the frequency of student 
comments relating science to engineering and technology. 


Ten percent of observations described a new realization 
that scientists sometimes need to build or invent the equip¬ 
ment to make measurements. These most likely stemmed 
from one film segment in which a researcher enthusiasti¬ 
cally described his laboratory. His research team had 
invented a scientific instrument and built it from 
"catalogue parts, like Tinker Toys —putting them together 
in a way that nobody has ever put them together before," 
as he described it. Several student comments clearly refer 
to this segment: 

I was surprised that the instruments they used in the 
lab were instruments they invented. (6th grade science 
student) 

It surprised me that scientists invent new machines 
from catalogue parts so that they can do exactly what 
they want done. (UG natural science student) 

Finally, in addition to these themes that address learn¬ 
ing about science, an eighth theme showed that students 
were also "learning science," gaining some understanding 
of the scientific content of the research study depicted in 
the film. While teaching scientific concepts was not a pri¬ 
mary goal of the film, it is gratifying to know that students 
did absorb some of the concepts referenced. Fourteen per¬ 
cent of observations, representing 17% of students, com¬ 
mented on a science concept. Many of these responded to 
an animation, shown twice in the film, of the collision 
between the Indian and Asian tectonic plates, as in this 
example: "Something that surprised me in the DVD was 
how India was just pushing itself into Asia. It is cool that 
the Earth can actually do that and form new land." Other 
students responded to the idea that rocks can be used to 
diagnose past climate and conditions. 

For the final assessment question, students were asked 
to study figures showing two different views of the scien¬ 
tific process. They were then asked to identify what ele¬ 
ments of each they saw in the film, and which version they 
thought best matched the film. Many student responses 
were not complete, answering only a portion of the two- 
part question. This may be due to the wording of the ques¬ 
tion in two parts or to a need for more explicit class discus¬ 
sion of the figures. So, while we cannot discern entirely 
what ideas led students to select one figure or another, we 
do see a trend toward a view of science more consistent 
with that held by scientists and philosophers (Reiff et ai, 
2002; Blachowicz, 2009). Over two thirds (69%) of students 
felt that Fig. 2, the inquiry wheel, best matched what they 
saw in the film. Of the 20% who felt the film best identified 
with the classical scientific method, many mentioned the 
iterative nature of the diagram, understanding that scien¬ 
tists have to "go back" if they do not answer their question 
the first time through. We interpret both types of responses 
as reflecting modest increases in the sophistication of 
students' understanding of the scientific process. 

Another 9% of students did not select either diagram 
and seemed not to comprehend what was being asked. 
Not having seen either diagram prior to the survey, it is 
possible that many were not sure of the meaning of the 
arrows and shapes showing the different processes. 
Indeed, one sixth-grade teacher reported that her students 
had difficulty with this question. Next time, she planned to 
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share and discuss the diagrams before showing the film, to 
prepare students to spot their components. 

DISCUSSION 

Student Responses to the Film Align with its Goals 

A basic but important interpretation of these data is 
that students take away messages from the film that are 
well aligned with its intentions and that reflect real 
changes in their ideas. Nearly 90% of student responses to 
question 2 can be classified as "learning about science," 
illustrating new or altered ideas about how science works. 
Students gain insights into the methods of science —the 
effort and number of samples required, the need for team 
members with varied expertise —and the creativity it 
requires. They see science as a type of work done by real 
people, perhaps even themselves in the future. They 
respond to the film's strong and deliberate framing of sci¬ 
ence as question-driven, and to the people they see in the 
film as different from stereotypes they have encountered 
earlier. In class discussion, they encounter additional ideas 
that may reinforce or challenge their own. 

The film set out to address students' views of both the 
intellectual and social processes of science. The seven 
themes within the broad "learning about science" category 
can be roughly grouped into these subsets: the themes 
we nicknamed as Methods, Questions, and Equipment 
(Table II) are more clearly related to the intellectual process 
of science, while themes entitled Collaboration, People, 
and Fun directly address the social processes. The theme 
we nicknamed Where includes elements of both: com¬ 
ments that some scientific investigations were conducted 
by making observations in the field, rather than by experi¬ 
ments in the laboratory, reflect understanding of an 
intellectual approach; while other comments noted that 
"scientists can work outside." Because the former idea 
dominated, we include this category with intellectual proc¬ 
esses. Using this grouping, 55% of all student comments 
address the intellectual processes of science, and 30% 
address social processes; the remaining 14% address 
science concepts. 

Moreover, these ideas expressed in student responses 
reflect actual changes in students' ideas and beliefs. Ques¬ 
tion 2 was deliberately phrased as "What surprised you or 
gave you new insight...?" in order to probe for changes in 
students' conceptions. Indeed, student responses often 
stand in direct opposition to the initial ideas they articu¬ 
lated or sketched for question 1: scientists are cooperative 
rather than solitary; interesting rather than "dorks"; not all 
male, white, or American. While faint praise at best, they 
reflect genuine changes in thinking: "There are people who 
are scientists but aren't totally wack." Science can be done 
in the field as well as the lab; science involves using cool 
equipment and inventing it oneself. "Something that gives 
me a new insight on science is that it involves creativity 
and imagination. When I think of science, I don't think of 
having to use your imagination and creativity!" 

Responses to question 3 are best read as reflecting a 
generally broadened view of the process of science. Some 
students detected a logical progression of hypotheses and 
observations in the film and thus preferred the linear scien¬ 
tific method diagram. A larger group recognized elements 
of a more complicated scientific process, as suggested by 


the inquiry wheel diagram. Both can be argued to be valid 
answers, but the choice forced by the phrasing of the ques¬ 
tion likely obscured students' real beliefs. Class discussion 
or teacher statements may have steered students toward 
the wheel diagram—in part because teachers' views too 
were expanded by the film and discussion, consistent with 
the film's educational goals. In teacher workshops, we find 
that the inquiry wheel provides teachers with useful lan¬ 
guage by which to link the classic scientific method com¬ 
monly portrayed in textbooks with a broader, more 
complex portrayal of science as conducted in practice. The 
results from question 3 are less useful for understanding 
student impact than are those from questions 1 and 2, but 
this item was pedagogically useful in generating discus¬ 
sion and encouraging students to consider aspects of both 
diagrams shown in the film. 

Developmental Differences in Student Responses 

We analyzed student responses by a broad grade level, 
separating middle school from high school and college stu¬ 
dents. The data are plotted in Fig. 3 using the same catego¬ 
ries listed in Table II; they reflect modest variation of 
responses by grade level. 

For example, middle school students more often 
remarked on certain social processes of science, such as col¬ 
laborating and working outside. High school students and 
undergraduates more often called out the intellectual proc¬ 
esses of science, noting aspects that were more abstract: 
methods that require effort and attention to detail, and the 
notion of science as answering questions. Middle school 
students' observations about investigations focused on 
pragmatic issues, such as the equipment that scientists 
used and invented. 

These may reflect real age-based differences in what 
students know already, what they learn from the film, 
and/or what interests them. Middle school is when stu¬ 
dents are typically identifying their interest—or lack of it — 
in science-oriented careers (Tai et al. r 2006). Thus, middle 
school students' interest in scientists' everyday work may 
reflect an openness to viewing the scientists in the film as 
possible future selves, while high school students are more 
likely to have already self-identified as science-oriented or 
not. High school students were also more likely to note 
surprise that science was fun —perhaps reflecting greater 
prevalence among this age group of a pre-existing expecta¬ 
tion that it was not fun. 

However, we also noted strong variations in the fre¬ 
quency of themes from class to class that likely reflect how 
teachers presented the film, their expectations for student 
writing, and the placement of the film within the overall 
course. This variability, and the fact that our sample of 350 
students is gathered from only 15 class sections taught by 
10 teachers, mean that we cannot reliably generalize about 
age differences in response. Still, it is clear that students 
across a wide span of ages, from grade 6 to college, can 
extract meaningful ideas from the film that are develop- 
mentally appropriate and consistent with the messages we 
sought to communicate. The film appears to offer multiple 
access points for a range of student audiences. 

Comparison with Other Studies 

Prior research has examined how science and 
scientists are presented in the media to children and adults 
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FIGURE 3. Age group differences in student responses to question 2. 


(Long and Steinke, 1996; Nisbet et al, 2002, and references 
therein). Such studies document some of the media images 
that this film directly sought to counter—portrayals of sci¬ 
entists as eccentric and antisocial, isolated, and elite 
explainers of knowledge. These are broad surveys of what 
is distributed to media viewers, rather than how they 
react. Other studies have correlated public attitudes to¬ 
ward science with general media consumption behaviors, 
such as television-watching and newspaper-reading, but 
cannot trace the influence of any particular medium or pro¬ 
gram (Nisbet et al, 2002; Miller, 2001). 

More targeted studies have sought to examine the 
impact of single science media programs by measuring 
varied outcomes for viewers, including understanding of 
specific science concepts (e.g.. Smith and Reiser, 1997), in¬ 
terest in science (e.g., Mares et al, 1999), behaviors (e.g., 
Glasgow et al, 2009), attitudes and beliefs (e.g., Penn et al, 
2003), or emotions (e.g., Barbas et al, 2009). For example, 
Fortner (1986) studied children's learning from a Cousteau 
Society documentary film about marine mammals using 
pre/post/delayed post-tests. The treatment group viewed 
the film, while the comparison group viewed a teacher's 
presentation of the film script. Children randomly assigned 
to treatment and comparison groups remembered the ma¬ 
terial equally well, but attitudes were more positively 
affected by the film. Our study is unusual in focusing on 
understanding of the nature and process of science as the 
measured student outcome. 

A recent report from the National Research Council 
points out, "Few studies speak to the impact of science- 
related [media] programs on the audience's understanding 
of science" (Bell et al, 2009, p. 275). Difficulties include 
determining whether the film reached its target audience, 
separating effects of the media program from the many 
other factors that influence people's ideas about science, 
and capturing data on impact from volunteer public audi¬ 
ences (Friedman, 2008; Von Seeker, 2001). It is clearly easier 
to detect the impact of multiple episodes of a TV series like 


Bill Nye the Science Guy (Rockman et al., 1996, cited in Bell 
et al, 2009) than to assess changes due to a single viewing 
experience. One advantage in the present study over stud¬ 
ies of broadcast media is that the main target audience can 
be identified and reached through their teachers. 

This study does not purport to establish or test student 
ideas about the nature of science, nor to compare interven¬ 
tions for influencing these ideas. Rather, it offers evidence 
about the impact of an educational media product devel¬ 
oped through a federally funded science E/O effort. The 
authentic assessment was provided as a teaching tool and 
used only secondarily for measuring impact. Our choices 
of sampling and data-gathering approaches were shaped 
by the project's goal to develop and distribute an educa¬ 
tional documentary film, rather than to conduct research. 
These choices in turn lead to some limitations in the evi¬ 
dence: the sample is one of convenience, and data are lack¬ 
ing about factors such as student gender and ethnicity that 
are known to affect student responses (Finson, 2002). The 
immediate pre-/post-test design means that we do not 
know the longevity of the changes in student ideas docu¬ 
mented. But by using our direct-to-teachers distribution 
approach, and by borrowing data analysis methods from 
educational research, we can collect data and draw conclu¬ 
sions about the film's impact on its end audience, second¬ 
ary students. Thus this study is a rare example of how 
audience learning —not just providers' E/O activity or 
audience satisfaction —can be measured, and it offers a 
model for evaluating the impact of E/O efforts by research 
projects and institutions. 

CONCLUSIONS AND FUTURE DIRECTIONS 

Many people have narrow or distorted views of 
science and scientists. These views will change only if they 
become aware of their own preconceptions and see alter¬ 
nate portrayals of science and scientists. Such experiences 
may be especially crucial for teachers, who are strong 
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influences on students' perceptions of science and scientists 
but who seldom have opportunities to meet real scientists 
(McDuffie, 2001; Cady and Rearden, 2007). Using an NSF- 
funded research study as a vehicle for communicating 
broader impacts of the research, we developed a documen¬ 
tary film to offer alternative views of science and scientists 
and collected evidence that helps to alter these perceptions. 
We believe the film was effective in accomplishing these 
goals because we drew on best practices from educational 
research and design to design and share the film. In partic¬ 
ular, we: 

• addressed a recognized gap in K-12 science curricu¬ 
lum materials, 

• communicated important ideas about how science 
works in an interesting manner, through the vehicle 
of a specific research study, 

• provided multiple points of access to these ideas for 
a wide range of audiences from grade 6 to adults, 

• involved scientists in ways that were appropriate, 
comfortable and interesting for them, and that drew 
on their expertise, 

• collaborated closely with professional filmmakers to 
create a visually stimulating, technically superior 
product, 

• engaged teachers personally in both making the film 
and preparing them to use it in the classroom, 

• supported teachers through professional develop¬ 
ment and classroom materials, 

• fostered student-active learning through pre/post 
questions that initiated individual reflection and 
group discussion, enhanced social learning, and gave 
teachers information about their own students' 
thinking, 

• gathered assessment data efficiently, using the same 
tool provided to teachers for classroom use, to ascer¬ 
tain what students were learning, and analyzed these 
data to assess evidence of impact. 

In conducting this E/O project, both process and 
product have proven to matter. We involved teachers in 
creating, disseminating, and learning from the product. 
Our combined experience has shown that the most- and 
best-used materials are those teachers have experienced, 
participated in, and created themselves. Some E/O projects 
have created websites full of curriculum on the content of a 
research project —but these are necessarily directed to a 
narrow group of teachers who teach that content in their 
classrooms, and may offer more information than even 
those teachers can cover in a standards-based curriculum. 
Here, teachers gain opportunities to learn and share with 
their students how science is done and who scientists are — 
something many teachers cannot otherwise provide to 
their students. E/O efforts like this one thus give scientists 
the opportunity to make a meaningful difference in K-12 
education. 

We are often asked for other films that show research 
projects in action — perhaps an example that shows more 
applied research, such as development of new drugs, to 
compare with the basic science focus of this study. And 
indeed, it is easy to imagine the greater power of a set of 
films that enable students to compare and contrast por¬ 
trayals of scientific research in action. We thus encourage 


other science E/O efforts to use an educational documen¬ 
tary as a tool when the scale of a project permits. Film is an 
efficient way for scientists to communicate with the public. 
Upward and Outward: Scientific Inquiry on the Tibetan Plateau 
reaches many more students and teachers than could be 
reached if the researchers invested the same amount of 
time in classroom visits. Those who do participate in class¬ 
room outreach are using it as a tool for their individual 
E/O activities. More broadly, we urge E/O projects to 
focus their efforts on building an understanding of the 
intellectual and social processes of science, which, we 
argue, can have a much greater influence on public views 
of science than does understanding of any particular 
scientific topic. 

Finally, we note that this study demonstrates, in a 
small way, that the education and outreach efforts of scien¬ 
tific organizations can, and should, take the same evi¬ 
dence-based approach used by the scientists whom they 
support. There are many reasons this is hard, but we owe 
it to our audiences and funders to both draw on, and 
contribute to, the evidence base about "what works" in 
E/O. We also owe it to ourselves to demonstrate that 
science E/O specialists occupy a real profession whose 
work is deserving of both respect and critical scrutiny. 
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